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Abstract 

We describe the development and initial evaluation of the efficacy of We Have 
Skills! (WHS), a video-based social skills instructional program for early el¬ 
ementary school students. The components of WHS were designed to be sci¬ 
entifically sound, maximally useful to elementary school teachers, and effec¬ 
tive in increasing students' social skills. Results from feasibility and social 
validity testing showed that teachers felt the program was easy to implement 
and highly recommended its use. The initial efficacy evaluation of WHS con¬ 
ducted with 70 classrooms randomly assigned to intervention and control 
conditions showed that teachers in the intervention group scored significantly 
higher on self-efficacy than teachers in the control group. Students in the in¬ 
tervention classrooms were rated significantly higher on key social skills by 
their teachers at posttest compared to students in the control group. Implica¬ 
tions for further testing of WHS are discussed, along with study limitations 
and recommendations for future research and practice. 
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M any students enter elementary school lacking basic social skills 
necessary to benefit from academic instruction (Hamre & Pi- 
anta, 2001; Kupersmidt, Bryant, & Willoughby, 2000; Lopez, Tarullo, 
Fomess, & Boyce, 2000; Ryan, Fauth, & Brooks-Gunn, 2006; Walker et 
al., 1998). Specific social skills, such as listening, staying on task, and 
following teacher instructions are associated with academic success 
(Alexander, Entwisle, & Dauber, 1994; Algozzine, Wang, & Violette, 
2011; Ervin, Schaughency, Goodman, McGlinchey, & Matthews, 2006; 
Horner et al., 2009; McIntosh, 2005; McIntosh, Homer, Chard, Boland 
& Good, 2006; Scott & Barrett, 2004; Wang & Algozzine, 2011). Peer 
related social skills are also associated with academic success and de¬ 
layed onset of harmful behaviors such as drug and alcohol use in later 
grades (Eddy, Reid, & Curry, 2002). 

Because of this association, explicitly teaching social skills to el¬ 
ementary students is strongly recommended (Cartledge & Milbum, 
1978; Gresham, 2004; Walker, Ramsey, & Gresham, 2004). Not only 
are social skills essential for students to benefit from classroom in¬ 
struction, they are critical for long-term life success (Elias, 2011). Stu¬ 
dents who engage in appropriate behaviors tend to have positive peer 
relationships (Ladd, 1999; Wentzel, Baker, & Russell, 2009) and better 
relationships with their teachers ultimately resulting in better aca¬ 
demic outcomes (Kim, Anderson, & Bashaw, 1968; Malecki & Elliott, 
2002; McClelland, Morrison, & Holmes, 2000; Swift & Spivack, 1969; 
Zins, Weissbert, Wang, & Walberg, 2004). Not surprisingly, students 
engaging in appropriate behavior receive more praise, more opportu¬ 
nities to respond, and less criticism (Good & Brophy, 1972). Students 
who receive more positive teacher attention outperform their peers 
who receive negative teacher attention (Baker, 2006). 

Ironically, with increasing pressure to prepare students to per¬ 
form well on high stakes academic tests, teachers often find little time 
to teach students the social skills they need to succeed academically 
(Anderson, 2009). In addition, teachers tend to receive little pre-service 
and in-service training in social skills instruction (Bromfield, 2006; 
Dobbins, Higgins, Pierce, Tandy, & Tincani, 2012; Priyadharshini & 
Robinson-Point, 2003; Reupert & Woodcock, 2010; Smart & Igo, 2010; 
Stoughton, 2007). As a result, they tend to resort to ineffective and 
unnecessarily punitive responses to student misbehavior in the class¬ 
room, rather than using effective social skills instruction (Gresham, 
Sugai, Horner, Quinn, & Mclnemy, 1998). 

Effective social skills instruction, like academic instruction, con¬ 
sists of distinct phases: acquisition, fluency building, and general¬ 
ization (Bullis, Walker, & Sprague, 2001; Walker, Schwarz, Nippold, 
Irvin, & Noell, 1994; White, 2005). During acquisition, instruction is 
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intended to develop the learner's conceptual understanding and 
performance of the skill, that is, accurate knowledge of the key com¬ 
ponents of the target skill and the ability to perform the discrete be¬ 
haviors associated with that skill. After initial acquisition, practice 
opportunities are presented and the learner is supported to build flu¬ 
ency through applying the skill in authentic settings (e.g., classroom, 
playground). Precorrection, feedback, and positive reinforcement are 
used to guide the learner through developing speed and accuracy 
when performing the skill (Colvin, Sugai, & Patching, 1993). Finally, 
the goal is to fully integrate the learned skill into the learner's behav¬ 
ioral repertoire so that its use will generalize to novel settings, persons 
and situations and sustain over time. Many social skills curricula fo¬ 
cus on skill acquisition, but neglect fluency building and generaliza¬ 
tion (Bullis et al., 2001). 

This paper describes the iterative development and presents 
outcomes from an initial evaluation of We Have Skills! (WHS; Mar¬ 
quez et al., 2012), a social skills program for early elementary school 
students that was specifically designed to (a) address the needs of the 
large number of students who begin school with weak or limited so¬ 
cial skills; (b) meet the needs of their teachers who lack the time, train¬ 
ing, and expertise to provide social skills instruction; and (c) be firmly 
rooted in the research on effective social skills instruction. We first 
describe the program components and provide an overview of their 
intended use. 

WHS comprises three separate, functional components. Module 
1 delivers student instructional materials. Module 2 offers an online 
tool for assessing student social behavior in response to instruction. 
Module 3 provides professional development materials covering de¬ 
livery of the instruction and more intensive forms of student support. 
See Figure 1 for an overview of the program components and their 
intended outcomes. The student instructional materials in Module 1 
consist of eight 5-minute video lessons: an introductory video focuses 
on what social skills are and why they are important to learn and 7 
lessons provide instruction in 7 core skills associated with school suc¬ 
cess and derived from the research (Hersh & Walker, 1983; Walker 
& Rankin, 1980; 1983). In the videos, the teacher, Mr. Lopez, and his 
students discuss each skill and interact with a "magic" chalk board 
that presents scenes, graphics, and realistic vignettes of students who 
are performing positive examples or negative examples of the skill. 
Each skill is modeled and student characters say why it is sometimes 
hard to do the skill (e.g., a friend distracts you, someone makes you 
feel bad, or your mind wanders), and difficult to get back on track. At 
the conclusion of the video, Mr. Lopez introduces an animal friend, an 
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Figure 1. Overview of We Have Skills! program components and their intended 
outcomes. 

animated character who represents the skill and who sings a catchy 
song about the skill. The featured interactions between the teacher 
and his students show students practicing the skill and inspire the 
viewers (students and teachers) to do the same. 

The videos teach students that appropriate behavior is a skill 
that can be acquired through learning and practice, and they also 
model how to teach behavioral skills for teachers. Mr. Lopez is 
shown to be positive, enthusiastic, and supportive. He validates the 
notion that social skills have to be learned in the classroom to sup¬ 
port school success and need to be practiced both in the classroom 
and in other settings such as the playground and at home. He clear¬ 
ly models how to present a skill to a student audience and how to 
provide opportunities to respond to students. Video instruction is 
supplemented with guided practice through role play and printed 
learning materials, such as tillable booklets, coloring pages, cards 
for complimenting others and expressing feelings such as remorse, 
and posters. Module 1 also includes a reinforcement system. When 
students use the social skills appropriately, the teacher gives stu¬ 
dents skill tickets featuring the animal friends. In addition, students 
self-manage with tally sheets, and parents learn of their students' 
social skills success through positive notes sent home with the stu¬ 
dent. When the teacher determines, based on his/her judgment, that 
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a student has mastered a social skill, the teacher gives the student a 
certificate of mastery. 

Module 1 was developed to respond to teachers' varying teach¬ 
ing styles and preferences, as well as time constraints. Based on Mr 
Lopez's demonstrations, teachers could adopt similar pacing in their 
instruction, interrupt the video lessons to provide opportunities to re¬ 
spond or to practice, or provide students with practice opportunities 
after the video lesson is complete. The instructional materials to be 
delivered by the teacher to the students were designed to be flexible 
enough to support teachers' varying approaches to instruction and 
varying needs of their students. 

Module 2 consists of the online Elementary Social Behavior As¬ 
sessment (ESBA; Marquez et al., 2012; Pennefather & Smolkowski, in 
review), an assessment tool delivered via the insPMT™. The ESBA 
allows teachers to screen entire classrooms as well as progress-mon¬ 
itor individual students on the extent to which they exhibit prosocial 
behaviors. Teachers rate their students on 12 items that are also de¬ 
rived from the research conducted by Walker and colleagues (Hersh 
& Walker, 1983; Walker & Rankin, 1980; 1983) and map onto the 7 core 
skills taught to students. 

The professional development materials contained in Module 3 
consist of a series of instructional videos and clearly designed lesson 
plans that guide teachers' implementation of the four components of 
the student instructional materials: (a) video lessons, (b) practice op¬ 
portunities, (c) songs, and (d) reinforcement systems. Specific steps 
are provided on how to facilitate appropriate practice with positive 
and negative examples, regular rehearsal with feedback, and ongoing 
review. In addition to the lesson plans. Module 3 contains additional 
practice materials for students who need more than universal instruc¬ 
tion (Tier II groups). Finally, Module 3 includes instruction on using 
the ESBA and znsPMT™ tool for universal screening, progress moni¬ 
toring, and data-driven decision-making. 

WHS purposefully blends the principles of three evidence-based 
practices that have emerged as critical to supporting students' social- 
emotional learning (Osher, Bear, Sprague, & Doyle, 2010; Zins et al., 
2004). These include positive behavior interventions and supports 
(PBIS; Lane, Kalberg, & Menzies, 2009; Sailor, Dunlap, Sugai, & Horn¬ 
er, 2009; Sugai et al., 2010), response to intervention (Rtl; Brown-Chid- 
sey & Steege, 2005; Gresham, 1991; Hawken, Vincent, & Schumann, 
2008), and explicit instruction (Archer & Hughes, 2011; Engelmann & 
Carnine, 1982; Hall, 2009). 

Research on PBIS in elementary schools has linked (a) defining 
expected behaviors, (b) teaching what those behaviors look like, (c) 
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rewarding students who engage in those behaviors, (d) consistently 
providing consequences for students who do not engage in those be¬ 
haviors, and (e) using continuously collected data for making deci¬ 
sions about students' behavioral support needs to improved social 
and academic outcomes (Bradshaw, Mitchell, & Leaf, 2010; Homer 
et al., 2009; Mitchell, Bradshaw, & Leaf, 2010). PBIS has been widely 
implemented in elementary schools (Horner et al., 2009). 

To meet all students' needs, it is recommended that social skills 
instruction, similar to academic instruction, follow the Rtl logic (Al- 
gozzine, Putnam, & Horner, 2010; Brown-Chidsey & Steege, 2005; 
Gresham, 1991). Universal instruction is implemented for all students 
at all times and, if implemented well, suffices to address the needs 
of the majority of students. Secondary (selective) and tertiary (indi¬ 
vidual student) instructional strategies are implemented as necessary 
to address the needs of students who remain unresponsive to univer¬ 
sal instruction (Hawken et al., 2008). Continuous progress monitoring 
helps teachers match the intensity of instruction and support(s) with 
student need (Sprague & Walker, 2010). 

Learning occurs effectively and efficiently when the principles 
of explicit instruction (El) are applied (Archer & Hughes, 2011; En- 
gelmann & Camine, 1982; Hall, 2009). El is a widely used approach 
to instructional design derived from more than 25 years of research 
and linked to student gains in academic achievement as well as self¬ 
esteem (Bereiter & Engelmann, 1966; Engelmann & Carnine, 1982; 
Watkins, 1988). The principles of El include (a) careful example selec¬ 
tion and sequencing, (b) modeling, (c) adequate pacing, (d) frequent 
opportunities to respond and practice, and (e) error correction. Se¬ 
lection and sequencing refer to arranging content from easy to dif¬ 
ficult, building on previously learned skills, and carefully scaffolding 
lessons to allow the learner to succeed. The teacher, or in WHS, Mr. 
Lopez, models each skill through examples as well as non-examples. 
Adequate pacing assures the learner stays engaged and constantly 
challenged according to his or her skill level. Continuous practice is 
provided through frequent opportunities to respond, and errors are 
immediately corrected to prevent learning misrules. Table 1 provides 
an overview of the key features of WHS and their derivation from 
evidence-based practice. 

Iterative Development of WHS 

In this paper, we focus primarily on Modules 1 and 3 of WHS, 
(i.e., the student instructional materials and the professional develop¬ 
ment materials). Development and psychometric testing of the assess¬ 
ment tool, the ESBA, contained in Module 2, is reported in Pennefather 
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Table 1 


Key Components of WHS and their Derivation 
from Evidence-Based Practice 


WHS feature 

Evidence-based practice 

7 core skills are clearly defined 

Clearly defined behavioral expectations 
(PBIS) 

Video lessons, songs, posters, and 
practice activities proactively teach core 
skills 

• Carefully sequenced video lessons 

• Child actors modeling skills 

• 8 six-minute videos, 1 per week 

• Built-in practice opportunities 

• Child actors demonstrate examples 

and non-examples of skills 

Proactive teaching of appropriate behav¬ 
iors (PBIS) 

• Effective selection and sequencing (El) 

• Modeling (El) 

• Appropriate pacing (El) 

• Opportunities to respond (practice) 

(El) 

• Error correction (El) 

Reinforcement system 

• Skill ticket pad 

• Student tally sheets 

• Happy notices (for parents) 

• Certificates of mastery 

Acknowledgement of appropriate be¬ 
haviors (PBIS) 

Teacher supports for addressing stu¬ 
dents' differing support needs: 

• Lesson plans for class-wide (univer¬ 

sal) instruction 

• Additional student activities for tar¬ 

geted (secondary) support 

Multi-tiered service delivery (Rtl) 

Online Elementary School Behavior 
Assessment 

Use of data for decision-making (PBIS, 

Rtl) 

Use of universal screening and progress¬ 
monitoring to assess effectiveness of 
lessons and individual student respon¬ 
siveness 

Progress-monitoring to inform instruc¬ 
tion and intervention (PBIS, Rtl) 


and Smolkowski (2013). The development of Module 1 of WHS was 
guided by close collaboration with consultants, information collected 
during multiple focus group meetings with teachers and students, 
social validity testing, and feasibility testing in general elementary 
school classrooms. 

Initial development. Initially, we intended to create a profession¬ 
al development program for teachers in order to increase their flu¬ 
ency in social skills instruction. Through numerous interviews and 
focus groups with teachers, however, we found that they needed far 
more than professional development. Teachers informed us that they 
were well aware of the need for social skills instruction, but that they 
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lacked the time, resources, and training to develop the systems nec¬ 
essary to define, teach, reward, enforce, and monitor the social be¬ 
haviors relevant to classroom instruction, let alone coordinate their 
approach across classrooms within a building. In response to this very 
clear message from teachers, we shifted our goal toward developing a 
program that offered teachers the opportunity to deliver short social 
skills lessons to their students with a minimum amount of effort. 

Our first task was to define the content for the short social skills 
lessons. In collaboration with our consultants, we defined seven core 
social skills based on the research conducted by Walker and col¬ 
leagues (Hersh & Walker, 1983; Walker & Rankin, 1980; 1983). Based 
on a survey of 56 descriptions of adaptive student behavior involv¬ 
ing 1,100 K-12 teachers. Walker and Rankin (1980) identified a subset 
of student behaviors that teachers associated with highly "teachable" 
and responsive students. These behaviors clustered into three areas: 
work habits, self-control and responsiveness to teacher demands, and 
positive relationships with peers (Foulks & Morrow, 1989). We used 
the content of these three domains as a guide to reduce the 56 adap¬ 
tive behaviors to seven core skills that would allow students to suc¬ 
ceed socially and academically. The seven skills WHS teaches are (a) 
listen, (b) ask for help, (c) follow directions, (d) do the best you can, (e) 
follow the rules, (f) work out strong feelings, and (g) get along. 

Based on teachers' and students' feedback to initial content 
matrices, we produced the student instructional materials consist¬ 
ing of eight video lessons: one introductory lesson about what so¬ 
cial skills are and why they are important and one lesson each on 
the seven identified skills. The videos consist of modeling vignettes, 
animal characters, animations, and songs. Samples of the skill vid¬ 
eos and the songs can be found at http://www.youtube.com/user/ 
irisedonline?feature=watch. 

After production of the videos and songs, our development 
team visited 3 elementary schools to gather information on what ma¬ 
terials and technologies are commonly available to teachers, and to 
observe current practice of promoting students' social skills acquisi¬ 
tion. Based on our findings, we produced the additional student ma¬ 
terials to supplement the video lessons: tillable skill practice booklets; 
skill posters; coloring pages; cards to help with complimenting, apol¬ 
ogizing, or expressing feelings; self-management tally sheets; take 
home "happy notices" that let parents know how well students are 
progressing with their behavioral skills; and certificates of mastery. 

Social validity and feasibility testing. We conducted a feasibility 
study to test whether WHS could be implemented in general elemen¬ 
tary school classrooms. Our goal was to assess the extent to which 
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teachers felt comfortable and prepared in using the student instruc¬ 
tional materials in their classrooms as well as the online assessment 
tool to screen and progress-monitor their students. Thirty-six kin¬ 
dergarten through third-grade teachers (31 female) from three school 
districts participated. Our project staff provided a two-hour inservice 
to introduce participants to the components of WHS, its theoretical 
background, and its intended use. Following the inservice training, 
teachers were asked to show their students video Lesson 1 (what so¬ 
cial skills are and why they are important), and two videos teaching 
specific skills. Data on teachers' sense of self-efficacy were collected 
pre- and post-implementation of WHS with the Teacher Sense of Effi¬ 
cacy Scale (TSES; Tschannen-Moran & Woolfolk Hoy, 2001). Teachers' 
comfort with using the online assessment tool was assessed at post¬ 
test with the Technology Acceptance Model (TAM; Davis, Bagozzi, & 
Warshaw, 1989). We also collected data on overall consumer satisfac¬ 
tion via a questionnaire specifically designed for this project. 

The TSES is a 12-item scale with three moderately correlated fac¬ 
tors: efficacy for student engagement, efficacy for instructional prac¬ 
tices, and efficacy for classroom management. In previous research, 
this instrument has shown strong internal reliability and construct 
validity (Tschannen-Moran & Woolfolk Hoy, 2001), and has been re¬ 
lated to a variety of outcomes including student achievement (Moore 
& Esselman, 1992; Ross, 1992), teacher planning and organization 
(Allinder, 1994), inclination to refer students to special education 
(Soodak & Podell, 1993), and commitment to teaching (Trentham, Sil¬ 
vern, & Brogdon, 1985). Because WHS includes a training module de¬ 
signed to support teachers in their delivery of social skills instruction, 
teacher sense of efficacy in providing this instruction was assessed. 

A paired t-test evaluating changes in teachers' ratings of self- 
efficacy found a significant increase in teacher self-efficacy from pre¬ 
test (mean = 7.2) to posttest (mean = 7.6), f(35) = 3.19, p < .003. We next 
examined whether the change in self-efficacy was related to teachers' 
comfort with using the online assessment tool, using an Analysis of 
Covariance (ANCOVA) including TAM scores at posttest. We found 
that teachers' TAM scores did not predict their changes in self-effica¬ 
cy, F(l, 34) = 1.67, p = .21. 

We found high consumer satisfaction with the WHS compo¬ 
nents. Overall, the teachers were very satisfied with the student ma¬ 
terials, with a mean consumer satisfaction score of 5.39 out of 6. The 
overall consumer satisfaction for the ESBA (4.0 out of 6) was signifi¬ 
cantly higher than neutral (3.5), f(34) = 2.62, p < .05. Overall, the teach¬ 
ers were quite positive about the program, with 100% stating they 
would recommend the program to other teachers (53% said strongly 
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recommend) and 100% said they were likely to use the program in 
the classroom (56% reported the highest likelihood). 

We then conducted a focus group with teachers who had partici¬ 
pated in this study to gain insights and ideas for completing the pro¬ 
fessional development materials intended to assist teachers in imple¬ 
menting WHS in their classrooms (Module 3). One clear report from 
teachers was that students loved the program and that high student 
favorability encouraged teachers to use the program extensively both 
during and after the study. For example, in the study debriefing ses¬ 
sions, we heard from one teacher that her students insisted on singing 
all seven skill songs at the start of every school day. Other teachers de¬ 
scribed the positive impact that the instruction had made on students 
with challenging behaviors. 

Evaluation of WHS 

The present study sought to formally evaluate WHS delivered as 
intended in classrooms and with a larger sample than the feasibility 
trial. This study evaluated the effect of WHS implementation on el¬ 
ementary students and their teachers and was driven by the following 
hypotheses: Compared to teachers who do not use the WHS program, 
teachers who use WHS will (a) demonstrate greater self-efficacy for 
achieving student engagement, instructional practices, and classroom 
management, and (b) see greater improvements in their students' 
classroom behaviors. 

Sample. Participant recruitment occurred in two school districts 
in California, and one each in Oregon and Washington. All K-3rd 
grade teachers in the districts were informed about the opportunity to 
take part in the study and asked to volunteer if interested. Interested 
teachers were informed about the study requirements and asked to 
provide informed consent. A total of 70 teachers (67 female) in K-3rd 
grade classrooms and their students (N = 1616) participated. Within 
each school, classrooms were randomly assigned to either the inter¬ 
vention condition (n = 37) or the control condition (n = 33) receiving 
business-as-usual. 

Demographics were collected from all teachers at pretest. Teach¬ 
er age ranged from 27 to 62, with an average age of 42.2 years. Teach¬ 
ing experience ranged from 1 to 38 years, with an average of 13.64 
years. Teachers' educational background included Bachelor degrees 
(n = 9), some post-secondary (n = 18), and graduate degrees (n = 40), 
with 3 teachers not reporting. Sixty-one of the teachers were Cauca¬ 
sian, 3 Asian, and 6 did not report. Four teachers identified as Hispan¬ 
ic. Of the 70 teachers, 15 taught kindergarten, 23 first grade, 17 second 
grade, and 15 third grade. 
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To capture a maximally diverse K-3 student population, the 
study was conducted in demographically diverse school districts. We 
were unable to collect demographic data at the student level. How¬ 
ever, across all K-3 classrooms in the participating schools, American 
Indian/Alaska Native student enrollment ranged from .1 to 2 % with 
an average of .5%, Asian/Pacific Islander enrollment ranged from 1 to 
29% with an average of 12%, Latino enrollment ranged from 11 to 47% 
with an average of 29%, African American enrollment ranged from 1 
to 18% with an average of 7%, White student enrollment ranged from 
13 to 77 % with an average of 42%, and Multiracial student enrollment 
ranged from 3 to 14% with an average of 8%. The percent of students 
eligible for free or reduced price lunch (PRL) ranged from 41 to 88%, 
with an average of 61%. 

Based on archival reports of school-wide behavioral support 
practices, 12 of the 17 schools participating in the study implemented 
SWPBIS. Generally SWPBIS focused on 3 school-wide behavioral ex¬ 
pectations: Be safe. Be respectful. Be responsible. In contrast to these 
generic school-wide expectations, WHS focused on teaching students 
specific social skills necessary to be successful in the classroom and 
build positive teacher and peer relationships. 

Procedures 

Experimental protocol. Teachers attended an in-service training 
presented by project staff during which the purpose of the study was 
described. Teachers completed informed consent and pretest mea¬ 
sures, after which they were randomly assigned to receive the WHS 
training and use the intervention in their classrooms, or to the busi- 
ness-as-usual control group who would receive access to the training 
and program at the completion of the evaluation. Both groups were 
then trained on how to use the online Elementary Social Behavior As¬ 
sessment (ESBA) to collect student data at pre- and posttest. The train¬ 
ing demonstrated how to (a) access the ESBA, (b) enter their students 
into the online system, and (c) complete the ESBA for each student. 
Control group participants were then excused, and the intervention 
participants were given an hour-long training on implementing WHS. 
Three weeks into the school year (after completing pretest ESBA as¬ 
sessments of their students) teachers in the intervention condition 
were instructed to begin using the WHS curriculum with their stu¬ 
dents. Teachers in the intervention condition taught one lesson a week 
(over eight weeks). Because of the short duration of the study, teach¬ 
ers were unable to conduct Tier II groups as part of the intervention; 
however teachers were trained to use ESBA data to identify students 
who might benefit from Tier II support. 
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Each week's activities included two core components: a five- to 
ten-minute discussion using discussion questions from the Lesson 
Plan and a five minute video lesson. In addition to these core com¬ 
ponents, teachers were encouraged to use practice activities such as 
songs (average of 30 seconds per skill), picture cards (2-4 minutes per 
skill), and role plays (1-5 minutes). The skills booklets were a take- 
home activity for students to complete with their parents (approxi¬ 
mately 5 minutes per skill). Reinforcement activities included asking 
students to complete the tally sheets (approximately 5 seconds), and 
handing out skills tickets, happy notices and certificates. Of the teach¬ 
ers in the intervention group, 34% reported spending one hour or less, 
16% reported spending 1-2 hours, 6% reported spending 2-3 hours, 
and 43% reported spending more than 3 hours on WHS program com¬ 
ponents during the 8-week duration of the study. 

Data collection. At the in-service training, teachers in both condi¬ 
tions completed demographics (including years of experience, educa¬ 
tion level, age, grade, and experience with internet technology) and 
pretest assessments of self-efficacy. Three weeks into the school year 
(and before using the WHS curriculum) participants in both groups 
completed an initial screening of their students' classroom behaviors 
using the ESBA. At the completion of the eight-week intervention 
period, teachers in both conditions completed a posttest assessment 
of their students' behaviors using the ESBA. Additionally, teachers 
completed the TSES. Intervention group teachers also completed mea¬ 
sures of social validity about the WHS program. 

Measures. To measure differences in teachers' perceptions of 
their skill mastery and their self-efficacy in social skills instruction 
due to use of the WHS program, teachers in both conditions complet¬ 
ed the TSES at pre- and posttest (see description above). To measure 
differences in teacher perceptions of student social behavior, teachers 
in both conditions completed the ESBA for all students in their class 
three weeks into the school year (but before the intervention teachers 
delivered the program to their students) and 8 weeks later, after the 
completion of the intervention. Teachers in the intervention condition 
completed a social validity measure at posttest. Each measure is de¬ 
scribed below. 

Elementary Social Behavior Assessment (ESBA). The ESBA is deliv¬ 
ered via the in’sPMT™, an online application for administering uni¬ 
versal screening and progress monitoring assessments. Teachers rate 
students on 12 behavioral items that map onto the seven social skills 
included in WHS. During universal screening, the teacher rates the 
extent to which each student engages in the 12 desirable behaviors 
on an intuitively understandable 3-point color-coded scale that corre- 
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Table 2 

Social Validity Items 



Mean 

Std Dev 

Overall, I was satisfied with the quality of this program. 

5.35 

.63 

I was satisfied with the quality of the information. 

5.32 

.63 

The training met my expectations. 

4.95 

.88 

I would recommend the program to other educators. 

5.35 

.82 

The program content was well organized. 

5.50 

.61 

It was easy to understand the ideas presented in the 
program. 

5.64 

.49 

I agree with the ideas presented in the program. 

5.73 

.51 

I am likely to use many of the strategies described in the 
program. 

5.35 

1.09 

The program was engaging. 

5.54 

.73 

It will be easy for me to implement this approach. 

5.32 

1.03 


sponds to the three-tiered student support models commonly used in 
interventions following the Rtl logic (Brown-Chidsey & Steege, 2005): 
green means "mastery," yellow means "needs improvement," and 
red means "cause for concern." Screening a classroom of 25 students 
takes approximately 15 to 20 minutes. Results of universal screenings 
are displayed in color-coded aggregations that allow teachers to easily 
and quickly determine (a) how many students in their class have mas¬ 
tered how many skills, (b) which individual students need additional 
instruction on which skills, and (c) which skills appear problematic 
for the majority of students. Students who need additional instruc¬ 
tion can be selected for progress monitoring. Progress monitoring 
is recorded on a 6-point scale to allow teachers to record more fine¬ 
grained incremental improvements in the student's skill acquisition. 
An automatically generated progress report shows the changes in rat¬ 
ings across time, indicating the student's progress. Teachers can use 
the report to assess whether to intensify, maintain, or fade instruction. 

Social Validity. Teachers in the intervention condition were asked 
about their experiences using the WHS program in their classroom. 
A six point Likert scale (ranging from 1 - strongly disagree to 6 - 
strongly agree), as well as open ended responses were used to assess 
teacher satisfaction with the WHS program at posttest (see Table 2 for 
individual items). 
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Results 

Data were analyzed using multi-level regression. The primary 
outcomes of interest were condition differences in teacher self-efficacy 
and student behaviors (nested under teacher). For teacher self-effica¬ 
cy we used an analysis of covariance (ANCOVA) examining condition 
differences at posttest, with pretest included as a covariate. Teachers 
in the two conditions did not differ significantly at pretest on years 
teaching (t(65) = -.76, p = .451), age (t(61) = 1.41, p = .163), gender (X 2 (l, 
N=70) = .48, p = .60), grade taught (X 2 (3, N=70) = 3.38, p = .34), educa¬ 
tion level (X 2 (2, N=68) = .32, p = .85), or race (X 2 (6, N=70) = 2.05, p = 
.92). They also did not differ on self-efficacy (f(68) = -.068, p = .499). 
Because we did not have demographic data at the student level, we 
were unable to conduct statistical equivalency tests. However, Table 
3 indicates that the overall K-3 populations in the intervention and 
control schools differed only minimally on race/ethnicity and FRL. 
Participating students did not differ at pretest on the ESBA, (F(l, 65) = 
2.10, p = .15). Because our study design randomized classrooms within 
school, school level variables such as SWPBIS implementation, should 
be equivalent across groups. 

For student behaviors, we ran a multilevel ANCOVA on the 
posttest ESBA nested under teacher and including pretest ESBA 
scores as a covariate. We also examined possible moderation of the 
teacher self-efficacy condition effect due to grade level of the class and 
mediation due to teacher experience and age. For the effect of con¬ 
dition on student behavior, we included teacher experience, teacher 
internet experience, and student grade level as potential moderators. 
We included teacher self-efficacy as a potential mediator on student 
behavior. Finally, we examined the qualitative social validity data col¬ 
lected from the teachers in the intervention condition. 

Teacher outcomes. A regression predicting teachers' ratings of self- 
efficacy at posttest as measured by the TSES, controlling for pretest 
TSES scores indicated that teacher self-efficacy improved significantly 
more for the intervention group (mean change = .54) than for the con¬ 
trol group (mean change = .15), F(l, 66) = 10.2, p = .002 (Cohen's d = 
.79). The effect of condition on self-efficacy did not depend on teacher 
gender (p = .915), grade taught (p = .092), age of teacher (p = .826), or 
years of experience (p = .985). 

Student outcomes. A multi-level regression was performed exam¬ 
ining condition effects on student behavior ratings controlling for de¬ 
pendency due to teacher. We included pre-intervention ESBA scores 
as a covariate in the analyses to account for error present at pretest, 
as well as some of the teacher error. Controlling for pretest, posttest 
ESBA scores for students in the WHS condition were significantly 
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Table 3 


Percent students enrolled in K-3rd grade classrooms in participating schools. 



Intervention 

Control 

American Indian/Alaska 
Native 

.375 

.65 

Asian/Pacific Islander 

12.75 

10.5 

Latino 

30.25 

28.5 

African-American 

7.25 

7.50 

White 

40.25 

33.75 

Multiracial 

8.0 

8.25 

Eligible for free or 
reduced price lunch (FRL) 

60.66 

61.46 


higher than scores for control students (f(64) = 3.4, p = .001). We found 
that students in the WHS condition improved by an average of 3.5 
points on the ESBA while students in the control condition improved 
by 1.7 points (Hedges' g = .27). 

Social validity outcomes. We found high social validity and usabil¬ 
ity results for the WHS materials. Overall, teachers were very satis¬ 
fied with the various student materials, with a mean social validity 
score of 5.4 out of 6 (see Table 2 for means and standard deviation 
of individual items). Teachers were quite positive about the overall 
program, with 100% stating they would recommend the program to 
other teachers (56% said strongly recommend) and 100% said they 
were likely to use the program in the classroom (59% reported the 
highest likelihood). 

Discussion 

Throughout the development of WHS, we faced considerable 
challenges to ensure that the program (a) was firmly grounded in 
evidence-based practice, (b) would have considerable appeal to early 
elementary school children, (c) would minimally infringe on teacher 
time, and (d) would be effective in improving student behavior. While 
the literature is clear on what constitutes best practices in social skills 
instruction (Gresham, 1991; Hawken et al., 2008; Lane et al., 2009), it 
is less clear on how to deliver social skills lessons to elementary aged 
students who are regularly exposed to specially designed entertain¬ 
ment products in theaters and television. Unless the materials can 
make a direct connection with children, instruction and student mo¬ 
tivation may be difficult to sustain. Our use of child actors, realistic 
vignettes, animal characters, and memorable songs successfully ap¬ 
pealed to elementary students. In many of the classrooms that tested 
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WHS, students demanded starting the day with the songs from the 
program. With each song lasting about 30 seconds, singing all 7 songs 
from the program would take a total of 3 minutes and 30 seconds. 

Given that time is a highly precious commodity for teachers, it 
was paramount that our social skills instructional program would take 
a minimum amount of time to implement. Unfortunately, many com¬ 
mercially available social skills curricula require that teachers spend 
significant time on staff development activities and implementing the 
intervention. For example, Poduska and colleagues (2008) reported 
that in order to implement the Good Behavior Game (Embry, 2002), 
teachers received 40 hours of training and support during one school 
year. Second Step (Committee for Children, 2012) requires that teach¬ 
ers use scripts to deliver lessons lasting a minimum of 25 minutes 
(Frey, Nolen, Edstrom, & Hirschstein, 2005). Teachers can distribute a 
25-minute lesson across one or several days per week. In their review 
of 33 commercially available social-emotional curricula, Berkowitz 
and Bier (2005) found that the majority of such commercially pub¬ 
lished social skills curricula rely on professional development as the 
primary pedagogical strategy. 

In contrast to programs emphasizing professional development, 
WHS provides practical tools to teach social skills. WHS is designed 
to allow teachers to quickly familiarize themselves with the pro¬ 
gram components and fluently implement them in their classroom. 
Available online, teachers spend approximately 1 hour familiarizing 
themselves with the program components and have access to fully 
developed lesson plans to deliver the social skills instruction to their 
students. Similarly, implementing WHS in the classroom takes a mini¬ 
mum of time away from academic instruction: Presenting a 5-minute 
video and engaging students in a brief discussion and practice oppor¬ 
tunities represents a highly efficient way of teaching social skills that 
are critical to students' school success. Consumer satisfaction survey 
data clearly indicated that teachers were delighted to have access to 
practical and efficient tools to teach social skills. 

Most importantly, however, a social skills program needs to ef¬ 
fectively change student behavior. Our evaluation study clearly in¬ 
dicated that students in classrooms that implemented WHS behaved 
better than students in classrooms that did not implement WHS. The 
positive associations between student prosocial behavior and (a) bet¬ 
ter academic outcomes (Alexander et al., 1994; Algozzine et al., 2011; 
Wang & Algozzine, 2011), (b) positive peer relationships (Ladd, 1999; 
Wentzel et al., 2009) and (c) improved student-teacher relationships 
(Kim et al., 1968; Malecki & Elliott, 2002; McClelland et al., 2000; Swift 
& Spivack, 1969; Zinset al., 2004) underscore the potential short-term 
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and long-term benefits students—and their teachers —can reap from 
using the WHS program. 

The initial success of WHS might be due partially to the pro¬ 
gram's inclusion of songs and practice opportunities to help students 
remember the target skills and integrate them into their behavioral 
repertoire. At the elementary level, song is commonly used to teach 
students academic content (e.g. the alphabet) and has been found 
an effective tool to retain information (Rainey & Larsen, 2002). WHS 
successfully transfers the use of songs from academic to social skills 
instruction to help students remember why the target skills are im¬ 
portant and what they look like. Because singing together has also 
been linked to positive social bonding (Hunter, Schellenberg, & 
Schimmack, 2008), the WHS songs not only serve as a memory aid for 
students, but also as a tool to build positive peer relationships. The 
fact that teachers reported that students insisted on beginning class 
by singing the WHS songs appears to support the value of including 
song into social skills instruction. 

Providing students with practice opportunities to reinforce their 
use of the target skills and promote their fluency in using them is 
equally important. The student materials included in WHS allow stu¬ 
dents to apply the learned skill in a variety of real contexts (e.g. using 
cards to compliment a peer) and actively engage them in the learning 
process. Social skills instruction thus moves beyond the conceptual 
mastery to fluency building and generalization across contexts. 

It is important to note, however, that our study did not collect 
data on treatment integrity. Participating teachers reported varying 
amounts of time they spent using the program during the course of 
the study. Based on this variation, it appears reasonable to assume 
that teachers integrated WHS in various ways into their classroom 
routines and activities. Close examination of treatment fidelity in fu¬ 
ture research will allow us to draw better conclusions about the pro¬ 
gram's ability to effect change in student behavior. 

Our initial evaluation of WHS clearly documented the program's 
social validity—a necessary prerequisite to its use—and its promise of 
effectiveness in changing student behavior. Puture research will focus 
on documenting the program's efficacy through a large scale random¬ 
ized control trial. Given teachers' strong interest in adopting WHS 
and its demonstrated effectiveness, we are confident that future stud¬ 
ies will show that the program is functionally related to improved 
student behavior. Puture research might also focus on expanding the 
program and adapting it for use in upper elementary and middle 
school classrooms. 
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Limitations 

Results from our initial evaluation of WHS must be interpreted 
against a number of limitations. First, the relatively small scope of our 
evaluation study resulted in small samples that limited generalizabil- 
ity of outcomes. Second, we were unable to control for school level 
factors, (e.g., overall enrollment, urbanicity, school-wide discipline 
and student support policies), teacher level factors (e.g. years of expe¬ 
rience) and student level factors (e.g. English learner status, disability 
status, race, gender) due to the small scope of our study. Thus, the in¬ 
ternal validity of our outcomes, i.e. the causality of our evidence, was 
limited. Demonstrations of a causal relationship between WHS imple¬ 
mentation and improved student behavior, taking into consideration 
teacher and student-level characteristics as moderators and school- 
level characteristics as mediators, will be the focus of future studies. 
Third, because of the limited duration of our study, we were unable 
to assess maintenance of skills across time. Students are able to reap 
the long-term benefits of social skills, such as fewer discipline refer¬ 
rals and better academic achievement, only if the effects of social skills 
instruction maintain over time. Future studies will include evaluation 
of the effectiveness of WHS across time. Fourth, the promise of WHS's 
ability to improve student behavior was measured with the ESBA, an 
instrument that was part of the intervention package and designed 
by the program developers. Therefore, interpretation of outcomes at 
present appears to be limited by the lack of a standardized measure. 
In addition, it is important to note that the ESBA reflects teacher re¬ 
ports of student behavior and is thus subject to potential teacher bias. 
Future studies should include measures of student behavior indepen¬ 
dent of teacher judgment, e.g. direct observations coded by impartial 
observers. However, Pennefather and Smolkowski's (2013) study on 
the relationship between ESBA and a standardized measure lends ini¬ 
tial support to our interpretation. Finally, our study did not include 
measures of treatment integrity. Future research on the effectiveness 
of WHS should assess treatment integrity through review of perma¬ 
nent products, or direct observations of program administration, in¬ 
cluding the number of minutes spent on each program component. 
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